Description

This PR adds support for moving Euler-Lagrange bubbles and various force models. Tracer particles that simply follow the background flow are implemented, as are Newton's 2nd Law particles that respond to body forces, pressure gradients, and drag. This PR is marked as a draft for now as there are several things that I need to complete before a proper merge can occur, but I wanted these changes to be visible to other contributors working on related features.

Type of change

• New feature

Testing

16 rank CPU versus 16 rank GPU

This test was ran using the examples/3D_lagrange_bubblescreen test case and compares the results for a 16 rank CPU simulation to a 16 rank GPU simulation. The visualization shows the following things:

1. The difference in pressure in the vertical plane, with darker colors corresponding to larger errors. There seems to be a reflection of some sort from the subsonic buffer outflow condition, and the acoustic source introduces a small amount of difference, though neither of these features was modified by this PR. (I intend to add a video with no bubbles in the domain to see if these artifacts still exist)
2. The difference in void fraction in the horizontal plane. The difference is 1e-38 everywhere in ParaView, which corresponds to bitwise identical results.
3. The bubbles for each simulation overlayed on top of each other. The bubbles from the CPU result are colored in blue while the bubbles from the GPU result are colored in white. The small bit of blue that is observed in the video is a result of the rendering spheres with a fixed number of facets.
4. The pressure field in the lower front quadrant as a surface render with darker colors indicating higher pressure.
5. The MPI domain decomposition as a black lattice.

Checklist

• I added or updated tests for new behavior
• I updated documentation if user-facing behavior changed

See the developer guide for full coding standards.

GPU changes (expand if you modified src/simulation/)

• GPU results match CPU results
• Tested on NVIDIA GPU or AMD GPU

Remaining items before merge

Must do

• Add regression tests for vel_model > 0 paths (tracer and Newton's 2nd Law) in toolchain/mfc/test/cases.py. The 2D_moving_lag_bubs and 3D_moving_lag_particles examples use adap_dt with parameters tuned for their full grid size and hang when reduced to 25x25 test grids. Dedicated test cases with appropriate parameters are needed.
• Add user-facing documentation in docs/documentation/case.md for new parameters (vel_model, drag_model, pressure_force, gravity_force, write_void_evol, input_path, charNz)

Should verify

• n_el_bubs_glb is rank-0 only after MPI_REDUCE — confirmed intentional (only rank 0 prints stats)
• nReal = 7 + 16*2 + 10*lag_num_ts magic number in s_initialize_particles_mpi — this buffer size must stay in sync with the pack/unpack loops. Consider computing it from named constants or adding a comment mapping each count to its field

Already fixed (by review commits)

Correctness fixes:

• 4th-order Lagrange interpolation: L(1) denominator xi(2)->xi(1), L(4) numerator xi(4)->xi(5) (m_bubbles_EL_kernels.fpp)
• case default in f_advance_step removed — dereferenced absent optional fVel/fPos args when called from EE path with adap_dt (m_bubbles.fpp)
• Restart write out-of-bounds: write loop now filters with particle_in_domain_physical to match allocation size (m_bubbles_EL.fpp)
• Vapor/gas property swap fixed in both example cases — all 10 bub_pp thermodynamic params were inverted (2D_moving_lag_bubs, 3D_moving_lag_particles)
• Missing $:END_GPU_PARALLEL_LOOP() in s_enforce_EL_bubbles_boundary_conditions (m_bubbles_EL.fpp)

GPU fixes:

• adap_dt_stop added to GPU private clause in m_bubbles_EE.fpp (race condition)

MPI fixes:

• s_mpi_reduce_int_sum #else branch for non-MPI builds (m_mpi_common.fpp)
• bubs_glb initialized to 0 before conditional MPI reduce (m_mpi_common.fpp)
• @:DEALLOCATE for p_send_buff, p_recv_buff, p_send_ids with allocated() guard (m_mpi_proxy.fpp)
• intent(in) added on lag_num_ts, nBub, pos, posPrev in MPI subroutines (m_mpi_proxy.fpp)

Code quality fixes:

• Duplicate hyper_cleaning removed from pre_process namelist (m_start_up.fpp)
• Bare 3 -> 3._wp in vel_model=3 force computation (m_bubbles.fpp)
• beta_vars indices documented: 1=void fraction, 2=d(beta)/dt, 5=energy source (m_constants.fpp)
• Uninitialized write_void_evol, pressure_force, gravity_force in pre_process (m_global_parameters.fpp)
• Stray sum.png removed from examples
• Pylint directives removed (replaced by ruff per lint: add pylint directive check and remove dead directives #1323)
• path variable shadowing fixed in cases.py

Test fixes:

• 2D_moving_lag_bubs and 3D_moving_lag_particles added to casesToSkip — these examples use adap_dt with parameters tuned for 200x200 grids. When reduced to 25x25, the CFL condition forces thousands of internal sub-steps, causing hangs. Stale golden files removed.

Review commit changelog (sbryngelson/Claude Code)

The following commits were added by @sbryngelson via Claude Code review to fix bugs and bring the PR to CI-passing state. All changes are to wilfonba's code; no new features were added.

52adde37 — Fix review bugs: Lagrange interpolation, GPU race, MPI, namelist

• L(1) denominator bug in 4th-order Lagrange interpolation (m_bubbles_EL_kernels.fpp:664): (xi(2) - xi(5)) corrected to (xi(1) - xi(5))
• L(4) numerator bug: (pos - xi(4)) corrected to (pos - xi(5)) — must exclude own index
• GPU race condition: adap_dt_stop added to private clause in m_bubbles_EE.fpp GPU parallel loop
• Non-MPI build fix: added #else; sum = var_loc to s_mpi_reduce_int_sum in m_mpi_common.fpp
• Uninitialized output: bubs_glb = 0 added before conditional MPI reduce in both MPI and non-MPI paths
• Duplicate namelist entry: removed duplicate hyper_cleaning from pre_process m_start_up.fpp
• Stray binary: removed examples/3D_moving_lag_particles/sum.png

506dccd2 — Fix optional arg UB, missing deallocates, missing intent

• Undefined behavior: removed case default in f_advance_step (m_bubbles.fpp) that wrote to absent optional args fVel/fPos when called from the EE path with adap_dt=T
• Memory leak: added @:DEALLOCATE for p_send_buff, p_recv_buff, p_send_ids in s_finalize_mpi_proxy_module
• Missing intent: added intent(in) on lag_num_ts in s_initialize_particles_mpi and nBub, pos, posPrev in s_add_particles_to_transfer_list
• Precision: 3 -> 3._wp in vel_model=3 force computation

f6adae3a — Fix dealloc crash on single-rank MPI

• Changed dealloc guard from if (bubbles_lagrange) to if (allocated(p_send_buff)) — the buffers are only allocated when num_procs > 1 (inside s_initialize_particles_mpi), so single-rank MPI runs crashed with "Attempt to DEALLOCATE unallocated"

fa2bc4e4 — Fix GPU directive, example physics, documentation

• Missing GPU directive: added $:END_GPU_PARALLEL_LOOP() after the locate-cell loop in s_enforce_EL_bubbles_boundary_conditions
• Physics bug: fixed systematic vapor/gas property swap in both 2D_moving_lag_bubs/case.py and 3D_moving_lag_particles/case.py — all 10 bub_pp thermodynamic parameters (gam_v/gam_g, M_v/M_g, k_v/k_g, cp_v/cp_g, R_v/R_g) were inverted vs the correct convention in 3D_lagrange_shbubcollapse
• Documentation: added inline comment documenting beta_vars indices in m_constants.fpp

99dbd4d0 — Fix restart write OOB and uninitialized lag_params

• Out-of-bounds write: s_write_restart_lag_bubbles allocated buffer with filtered count (bub_id) but wrote loop iterated over total count (n_el_bubs_loc). Added particle_in_domain_physical filter to the write loop using a separate counter i
• Uninitialized variables: added write_void_evol = .false., pressure_force = .false., gravity_force = .false. to pre_process m_global_parameters.fpp default initialization

d030b8a4 — Remove pylint directives

• Removed 2 # pylint: disable comments from cases.py (replaced by ruff per upstream lint: add pylint directive check and remove dead directives #1323)

c0f63238 — Formatting and lint cleanup after rebase

• Auto-formatted files after rebase onto master
• Fixed path variable shadowing lint error in cases.py (renamed to case_path)
• Removed === separator comment that new source linter flags

35a6d5bd — Skip hanging example tests

• Added 2D_moving_lag_bubs and 3D_moving_lag_particles to casesToSkip in cases.py
• Root cause: these examples use adap_dt=T with dt=1 on domain [-2000,2000]. When the test framework reduces the grid from 200x200 to 25x25, cell size increases from 20 to 160, but sound speed (~1500 m/s) stays the same. The adaptive stepper gets stuck in thousands of sub-steps trying to satisfy CFL, causing an infinite hang.
• Removed stale golden files for both tests

AI code reviews

Reviews are not triggered automatically. To request a review, comment on the PR:

• @coderabbitai review — incremental review (new changes only)
• @coderabbitai full review — full review from scratch
• /review — Qodo review
• /improve — Qodo code suggestions